stream: Readable.concat util method - #40075
Conversation
| @@ -2077,6 +2077,27 @@ Calling `Readable.from(string)` or `Readable.from(buffer)` will not have | |||
| the strings or buffers be iterated to match the other streams semantics | |||
| for performance reasons. | |||
|
|
|||
| ### `stream.Readable.concat(streams, [options])` | |||
|
|
|||
| * `streams` {Readable[]|Iterable[]|AsyncIterable[]} List of readable streams or iterator of readable stream generator | |||
There was a problem hiding this comment.
| * `streams` {Readable[]|Iterable[]|AsyncIterable[]} List of readable streams or iterator of readable stream generator | |
| * `streams` {Readable[]|Iterable[]|AsyncIterable[]} List of readable streams or | |
| iterator of readable stream generator. |
Every line must be 80 characters or less in length and this line should end with a period.
| By default, `Readable.concat()` will set `options.objectMode` to `false`, unless | ||
| this is explicitly opted out by setting `options.objectMode` to `true`. |
There was a problem hiding this comment.
| By default, `Readable.concat()` will set `options.objectMode` to `false`, unless | |
| this is explicitly opted out by setting `options.objectMode` to `true`. | |
| By default, `Readable.concat()` will set `options.objectMode` to `false`, | |
| unless this is explicitly opted out by setting `options.objectMode` to `true`. |
Every line must be 80 characters or less in length.
| A utility method to concatenate multiple readable streams to single one. The main readable stream will consume the | ||
| list of readable streams in order |
There was a problem hiding this comment.
| A utility method to concatenate multiple readable streams to single one. The main readable stream will consume the | |
| list of readable streams in order | |
| A utility method to concatenate multiple readable streams to single one. The | |
| main readable stream will consume the list of readable streams in order. |
Every line must be 80 characters or less in length and this line must end with a period.
| } | ||
|
|
||
| Readable.concat = function(streams, opts) { | ||
| return Readable.from(streamSeriesIterator(streams), {objectMode: false, ...opts}); |
There was a problem hiding this comment.
| return Readable.from(streamSeriesIterator(streams), {objectMode: false, ...opts}); | |
| return Readable.from(streamSeriesIterator(streams), | |
| { objectMode: false, ...opts }); |
Lint.
| } | ||
| } | ||
|
|
||
| Readable.concat = function(streams, opts) { |
There was a problem hiding this comment.
To make Readable.concat.length equal to 1:
| Readable.concat = function(streams, opts) { | |
| Readable.concat = function(streams, opts = undefined) { |
| for await (const chunk of stream) { | ||
| yield chunk; | ||
| } |
There was a problem hiding this comment.
Would that have the same effect?
| for await (const chunk of stream) { | |
| yield chunk; | |
| } | |
| yield* stream; |
There was a problem hiding this comment.
This works fine and looks better. Thanks
e03b91f to
41bb9f4
Compare
|
cc @nodejs/streams |
mcollina
left a comment
There was a problem hiding this comment.
Thanks for the PR!
Would you mind to add a few unit test for this feature?
I think if one of the "waiting" streams errors all should be destroyed, similar to how pipeline works.
|
|
||
| Readable.concat = function(streams, opts = undefined) { | ||
| return Readable.from(streamSeriesIterator(streams), | ||
| { objectMode: false, ...opts }); |
There was a problem hiding this comment.
This doesn’t support old streams. See pipeline for how we fix that.
|
@mcollina I quickly lookup and only we need destroy remaining readable streams if it is an async function * streamSeriesIterator(streams) {
let index = 0;
try {
for await (const stream of streams) {
index++;
yield* stream;
}
} finally {
if (streams instanceof Array) {
streams.slice(index).forEach(stream => stream.destroy());
}
}
}or async function * streamSeriesIterator(streams) {
let index = 0;
try {
for await (const stream of streams) {
index++;
yield* stream;
}
} catch (err) {
if (streams instanceof Array) {
streams.slice(index).forEach(stream => stream.destroy(err));
}
throw err;
}
} |
|
As an alternative suggestion... consider adding this utility to the import { concat, arrayBuffer } from 'node:stream/consumers';
const readable1 = new ReadableStream();
const readable2 = new ReadableStream();
const readable3 = new stream.Readable();
for await (const chunk of concat(readable1, readable2, readable3)) {
console.log(chunk);
}
// Let's say you want all of the contents in a single ArrayBuffer...
await arrayBuffer(concat(readable1, readable2, readable3));The return value would be just an async iterator. If someone wanted a const readable = new ReadableStream({
async pull(c) {
for await (const chunk of concat(readable1, readable2, readable3))
c.enqueue(chunk);
c.close();
}
}); |
|
@tugrul unfortunately #40075 (comment) does not work for node streams. They can error at any time even if they are not being consumed, you need to add on('error') handlers. |
|
Hello, I'm doing some backlog hygiene on all PRs related to networking and streaming modules. There hasn't been any activity here for quite a while so I'm going to mark it Thank you! |
|
This issue/PR was marked as stalled, it will be automatically closed in 30 days. If it should remain open, please leave a comment explaining why it should remain open. |
41bb9f4 to
2e226ec
Compare
2e226ec to
b2ab59a
Compare
b2ab59a to
9bf4fb7
Compare
9bf4fb7 to
66e2a87
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #40075 +/- ##
==========================================
- Coverage 90.15% 90.14% -0.02%
==========================================
Files 744 745 +1
Lines 242517 242917 +400
Branches 45688 45782 +94
==========================================
+ Hits 218642 218968 +326
- Misses 15358 15431 +73
- Partials 8517 8518 +1
🚀 New features to boost your workflow:
|
66e2a87 to
32aeb80
Compare
32aeb80 to
7fc582e
Compare
7fc582e to
1a78e41
Compare
1a78e41 to
7891de2
Compare
7891de2 to
544f96c
Compare
544f96c to
b3e3711
Compare
b3e3711 to
4c4da65
Compare
4c4da65 to
7d3fbad
Compare
7d3fbad to
a9449d1
Compare
a9449d1 to
82f0dd4
Compare
82f0dd4 to
2097cec
Compare
2097cec to
c10dd5f
Compare
Adds stream.concat(sources[, options]), which reads a sequence of readable sources as a single async iterator. Each source is consumed to completion before the next is started. Returning an iterator rather than a Readable avoids having to pick an objectMode on the caller's behalf: a concatenation of byte streams and one of object streams want opposite defaults, and only the sources know which case applies. Callers that want a stream can wrap the result with Readable.from(). Sources are guarded as they are acquired rather than as they are read. A Readable is live from construction, so a queued source that fails while an earlier one is still draining would otherwise emit 'error' with no listener attached and take down the process. concat() attaches a handler to every source it holds a reference to, reports the first failure, and destroys the rest. Because a collection that already holds live streams and a producer that creates one per iteration are indistinguishable, the input protocol selects the behavior: a sync iterable is drained and guarded up front, an async iterable is pulled one source at a time. Factories stay inert until reached, and options.lazy overrides the default where it guesses wrong. Refs: nodejs#40075 Assisted-By: Claude/Opus 5 Signed-off-by: Tuğrul Topuz <tugrultopuz@gmail.com>
c10dd5f to
d019ad6
Compare
There are many npm modules for this functionality and I think it is a common requirements because I see 452k weekly downloads of multistream npm package.